The Pitfalls of Using Open Data to Develop Deep Learning Solutions for COVID-19 Detection in Chest X-Rays.

Since the emergence of COVID-19, deep learning models have been developed to identify COVID-19 from chest X-rays. With little to no direct access to hospital data, the AI community relies heavily on public data comprising numerous data sources. Model performance results have been exceptional when training and testing on open-source data, surpassing the reported capabilities of AI in pneumonia-detection prior to the COVID-19 outbreak. In this study impactful models are trained on a widely used open-source data and tested on an external test set and a hospital dataset, for the task of classifying chest X-rays into one of three classes: COVID-19, non-COVID pneumonia and no-pneumonia. Classification performance of the models investigated is evaluated through ROC curves, confusion matrices and standard classification metrics. Explainability modules are implemented to explore the image features most important to classification. Data analysis and model evalutions show that the popular open-source dataset COVIDx is not representative of the real clinical problem and that results from testing on this are inflated. Dependence on open-source data can leave models vulnerable to bias and confounding variables, requiring careful analysis to develop clinically useful/viable AI tools for COVID-19 detection in chest X-rays.

Development and validation of multivariable machine learning algorithms to predict risk of cancer in symptomatic patients referred urgently from primary care: a diagnostic accuracy study.

OBJECTIVES: To develop and validate tests to assess the risk of any cancer for patients referred to the NHS Urgent Suspected Cancer (2-week wait, 2WW) clinical pathways. SETTING: Primary and secondary care, one participating regional centre. PARTICIPANTS: Retrospective analysis of data from 371 799 consecutive 2WW referrals in the Leeds region from 2011 to 2019. The development cohort was composed of 224 669 consecutive patients with an urgent suspected cancer referral in Leeds between January 2011 and December 2016. The diagnostic algorithms developed were then externally validated on a similar consecutive sample of 147 130 patients (between January 2017 and December 2019). All such patients over the age of 18 with a minimum set of blood counts and biochemistry measurements available were included in the cohort. PRIMARY AND SECONDARY OUTCOME MEASURES: sensitivity, specificity, negative predictive value, positive predictive value, Receiver Operating Characteristic (ROC) curve Area Under Curve (AUC), calibration curves RESULTS: We present results for two clinical use-cases. In use-case 1, the algorithms identify 20% of patients who do not have cancer and may not need an urgent 2WW referral. In use-case 2, they identify 90% of cancer cases with a high probability of cancer that could be prioritised for review. CONCLUSIONS: Combining a panel of widely available blood markers produces effective blood tests for cancer for NHS 2WW patients. The tests are affordable, and can be deployed rapidly to any NHS pathology laboratory with no additional hardware requirements.

A data flow process for confidential data and its application in a health research project.

BACKGROUND: The use of linked healthcare data in research has the potential to make major contributions to knowledge generation and service improvement. However, using healthcare data for secondary purposes raises legal and ethical concerns relating to confidentiality, privacy and data protection rights. Using a linkage and anonymisation approach that processes data lawfully and in line with ethical best practice to create an anonymous (non-personal) dataset can address these concerns, yet there is no set approach for defining all of the steps involved in such data flow end-to-end. We aimed to define such an approach with clear steps for dataset creation, and to describe its utilisation in a case study linking healthcare data. METHODS: We developed a data flow protocol that generates pseudonymous datasets that can be reversibly linked, or irreversibly linked to form an anonymous research dataset. It was designed and implemented by the Comprehensive Patient Records (CPR) study in Leeds, UK. RESULTS: We defined a clear approach that received ethico-legal approval for use in creating an anonymous research dataset. Our approach used individual-level linkage through a mechanism that is not computer-intensive and was rendered irreversible to both data providers and processors. We successfully applied it in the CPR study to hospital and general practice and community electronic health record data from two providers, along with patient reported outcomes, for 365,193 patients. The resultant anonymous research dataset is available via DATA-CAN, the Health Data Research Hub for Cancer in the UK. CONCLUSIONS: Through ethical, legal and academic review, we believe that we contribute a defined approach that represents a framework that exceeds current minimum standards for effective pseudonymisation and anonymisation. This paper describes our methods and provides supporting information to facilitate the use of this approach in research.

Estimated impact of the COVID-19 pandemic on cancer services and excess 1-year mortality in people with cancer and multimorbidity: near real-time data on cancer care, cancer deaths and a population-based cohort study.

OBJECTIVES: To estimate the impact of the COVID-19 pandemic on cancer care services and overall (direct and indirect) excess deaths in people with cancer. METHODS: We employed near real-time weekly data on cancer care to determine the adverse effect of the pandemic on cancer services. We also used these data, together with national death registrations until June 2020 to model deaths, in excess of background (pre-COVID-19) mortality, in people with cancer. Background mortality risks for 24 cancers with and without COVID-19-relevant comorbidities were obtained from population-based primary care cohort (Clinical Practice Research Datalink) on 3 862 012 adults in England. RESULTS: Declines in urgent referrals (median=-70.4%) and chemotherapy attendances (median=-41.5%) to a nadir (lowest point) in the pandemic were observed. By 31 May, these declines have only partially recovered; urgent referrals (median=-44.5%) and chemotherapy attendances (median=-31.2%). There were short-term excess death registrations for cancer (without COVID-19), with peak relative risk (RR) of 1.17 at week ending on 3 April. The peak RR for all-cause deaths was 2.1 from week ending on 17 April. Based on these findings and recent literature, we modelled 40% and 80% of cancer patients being affected by the pandemic in the long-term. At 40% affected, we estimated 1-year total (direct and indirect) excess deaths in people with cancer as between 7165 and 17 910, using RRs of 1.2 and 1.5, respectively, where 78% of excess deaths occured in patients with ≥1 comorbidity. CONCLUSIONS: Dramatic reductions were detected in the demand for, and supply of, cancer services which have not fully recovered with lockdown easing. These may contribute, over a 1-year time horizon, to substantial excess mortality among people with cancer and multimorbidity. It is urgent to understand how the recovery of general practitioner, oncology and other hospital services might best mitigate these long-term excess mortality risks.